Could not prefetch mysql_grant provider 'mysql': Execution of '/usr/bin/mysql

[Mikhail Samoylov]

Deploy cluster with 3 nodes with db, keystone, rabbit, horizon

Steps for reproduce:
        1. Create cluster
        2. Add 3 nodes with controller role
        3. Add 3 nodes with database, keystone, rabbit,
           horizon
        4. Add 1 compute and cinder
        5. Verify networks
        6. Deploy the cluster
        7. Verify networks
        8. Run OSTF
Actual result:
Deploy failed

Expected result:
Deploy finished without any errors and OSTF passed

CI job:
https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.plugins.thread_2_separate_services/120/console
Snapshot:
https://drive.google.com/open?id=0B2SenDuhfXPlaVB6N2VoVWZiSEE
Error from astute:
http://paste.openstack.org/show/505890/
Error from puppet log:
http://paste.openstack.org/show/505892/

ISO: 404

[Vladimir Kuklin (vkuklin)]

According to the first log, database backend was not up when first database users creation commands were run. It could have happened due to our manifests trying to access the DB via virtual IP which resides on an haproxy node which has no yet marked any of the DB backends as UP. We need to:
1) either create root user through local socket on the primary database node
or
2) wait for haproxy backend to get into UP state.

[Vladimir Kuklin (vkuklin)]

The test separate_rabbit_service_add_delete_node.separate_rabbit_service_add_delete_node failed for a separate reason: it could not run network verification due to node with id '1' getting offline.

"AssertionError: Task 'verify_networks' has incorrect status. error != ready, 'Method verify_networks. Network verification not available because nodes ["1"] not available via mcollective."