heat-api and heat-engine were running with wrong config on node-43 (on other two controllers services was running with right configuration). Therefore when heat started used the following messages appeared in logs:
/var/log/upstart/heat-api.log http://paste.openstack.org/show/242569/
/var/log/upstart/heat-engine.log http://paste.openstack.org/show/242570/
But heat-engine is running via corosync, therefore when it was broken, corosunc started up it again I believe.
I restarted heat-api services on each controller node at ~ 28 10:29. After that heat-api on node-43 works fine:
reproduced the issue on scale lab: e6fcc88c3184332 1759ec45d5 6d2829bea9912d0 baa3be5033 0b8b2a4a7eab658 6583ffaf33 d336c4cc253606d 4dd0d59a60 1c3fd5b139c966d 680d81e0a5 fuelclient_ sha: 38765563e1a7f14 f45201fd47cf507 393ff5d673
api: '1.0'
astute_sha: b09729c64b695b2
auth_required: true
build_id: 2015-05-16_20-55-26
build_number: '425'
feature_groups:
- mirantis
fuel-library_sha: 1645fe45f226cdd
fuel-ostf_sha: 9ce180074908178
fuelmain_sha: 0e970647a83d9a7
nailgun_sha: 076566b5df37f68
openstack_version: 2014.2.2-6.1
production: docker
python-
release: '6.1'
Baremetal, Ubuntu, IBP,HA, Neutron- vlan,Ceph- all,Nova- debug,Nova- quotas, 6.1_425
Controllers:3 Computes:200
During rally light test (rally with small number iterations) all heat tests were failed. paste.openstack .org/show/ 242458/ paste.openstack .org/show/ 242459/
rally.log:
http://
haproxy:
http://
heat-api and heat-engine were running with wrong config on node-43 (on other two controllers services was running with right configuration). Therefore when heat started used the following messages appeared in logs: upstart/ heat-api. log paste.openstack .org/show/ 242569/ upstart/ heat-engine. log paste.openstack .org/show/ 242570/
/var/log/
http://
/var/log/
http://
But heat-engine is running via corosync, therefore when it was broken, corosunc started up it again I believe.
I restarted heat-api services on each controller node at ~ 28 10:29. After that heat-api on node-43 works fine:
heat logs and puppet logs from node-43 are here: http:// mos-scale- share.mirantis. com/node- 43-339. tar.gz