Comment 10 for bug 1454794

reproduced the issue on scale lab:
api: '1.0'
astute_sha: b09729c64b695b2e6fcc88c31843321759ec45d5
auth_required: true
build_id: 2015-05-16_20-55-26
build_number: '425'
feature_groups:
- mirantis
fuel-library_sha: 1645fe45f226cdd6d2829bea9912d0baa3be5033
fuel-ostf_sha: 9ce1800749081780b8b2a4a7eab6586583ffaf33
fuelmain_sha: 0e970647a83d9a7d336c4cc253606d4dd0d59a60
nailgun_sha: 076566b5df37f681c3fd5b139c966d680d81e0a5
openstack_version: 2014.2.2-6.1
production: docker
python-fuelclient_sha: 38765563e1a7f14f45201fd47cf507393ff5d673
release: '6.1'

Baremetal,Ubuntu,IBP,HA,Neutron-vlan,Ceph-all,Nova-debug,Nova-quotas,6.1_425
Controllers:3 Computes:200

During rally light test (rally with small number iterations) all heat tests were failed.
rally.log:
http://paste.openstack.org/show/242458/
haproxy:
http://paste.openstack.org/show/242459/

heat-api and heat-engine were running with wrong config on node-43 (on other two controllers services was running with right configuration). Therefore when heat started used the following messages appeared in logs:
/var/log/upstart/heat-api.log
http://paste.openstack.org/show/242569/
/var/log/upstart/heat-engine.log
http://paste.openstack.org/show/242570/

But heat-engine is running via corosync, therefore when it was broken, corosunc started up it again I believe.

I restarted heat-api services on each controller node at ~ 28 10:29. After that heat-api on node-43 works fine:

heat logs and puppet logs from node-43 are here: http://mos-scale-share.mirantis.com/node-43-339.tar.gz