Inconsistent Floating IP pool list response during deployment

[Nikita Konovalov]

Found during the following bug investigation. https://bugs.launchpad.net/mos/+bug/1647286

While creating the set of default templates Sahara fails on the Floating IP pool validation.
Sahara checks that a given Floating IP Pool exists by fetching a list of Floating IP Pools from nova client. However it gets different responses in sequential requests.

First request returns with a correct response containing the network that Sahara is looking for
http://paste.openstack.org/show/592388/

Sahara continues validating other templates and sends the *same* request which returns an empty list of networks.
http://paste.openstack.org/show/592389/

The validation fails causing the failure of the whole deployment.

(Log entries are taken from var/log/sahara/sahara-api.log on node-3-10.109.2.4 See the snapshot attached to the bug https://bugs.launchpad.net/mos/+bug/1647286)

[Nikita Konovalov (nkonovalov)]

Replaying Sahara deployment puppet manifests 10 minutes later manually does not reproduce the issue and the templates are created and validated successfully.

This most likely means that there is a race condition in the Neutron backend or a split-brain in the database.