Deployment of nine-node env failed on step glance-manage sync db

[Tatyanka]

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.1"
  api: "1.0"
  build_number: "92"
  build_id: "2015-02-01_22-55-01"
  nailgun_sha: "c0932eb5c2aa7fd1e13a999cb1b4bf5aff101c3b"
  python-fuelclient_sha: "2ea7b3e91c1d2ff85110bf5abb161a6f4e537358"
  astute_sha: "ed5270bf9c6c1234797e00bd7d4dd3213253a413"
  fuellib_sha: "c5e4a0410ba66f9e9911f62b3b71c0b9c29aed6e"
  ostf_sha: "c9100263140008abfcc2704732e98fbdfd644068"
  fuelmain_sha: ""

 """Deploy cluster with separate roles on 9 nodes in HA mode with GRE

        Scenario:
            1. Create cluster
            2. Add 3 nodes as controllers
            3. Add 2 nodes as compute
            4. Add 1 node as cinder and 1 as mongo
            5. Add 2 nodes as ceph
            6. Turn on Sahara and Ceilometer
            7. Deploy the cluster
            8. Check networks and OSTF

Actual status:
Deployment failed(see - node-1 in snapshot)
When we try to execute in puppet command glance - manage sync db - mysql go shutdown in the normal way
As result some of db migartion was not synced
see glance-manage log at 2015-02-03 02:28:09.874
http://paste.openstack.org/show/166300/

at the same time in mysql log
paste.openstack.org/show/166301/

[Bogdan Dobrelya (bogdando)]

"When we try to execute in puppet command glance - manage sync db - mysql go shutdown in the normal way
As result some of db migartion was not synced" it sounds like a major issue caused by race condition, raised to high

[Bogdan Dobrelya (bogdando)]

The deployment failed as the only mysql backend went down in the middle of primary-controller deploy:
2015-02-03T10:29:20.877547+00:00 alert: Server mysqld/node-1 is DOWN, reason: Layer7 timeout, check duration: 10000ms. 0 active and 0 backup servers left. 3 sessions active, 0 requeued, 0 remaining in queue.

And the failure was caused by some OS issues not related to the Openstack deployment:
./node-1.test.domain.local/kernel.log
2015-02-03T10:29:21.278381+00:00 err: INFO: task jbd2/dm-1-8:300 blocked for more than 120 seconds.
2015-02-03T10:29:21.278381+00:00 err: INFO: task auditd:906 blocked for more than 120 seconds.
2015-02-03T10:29:21.281753+00:00 err: INFO: task rs:action 1 que:15443 blocked for more than 120 seconds.
2015-02-03T10:29:21.281753+00:00 err: INFO: task mysqld:3381 blocked for more than 120 seconds.
2015-02-03T10:29:21.281753+00:00 err: INFO: task mysqld:21595 blocked for more than 120 seconds.
2015-02-03T10:29:21.765535+00:00 err: INFO: task neutron-dhcp-ag:26321 blocked for more than 120 seconds.
2015-02-03T10:29:21.765535+00:00 err: INFO: task lrmd:21822 blocked for more than 120 seconds.