heat-api is turned off after a while

[Anastasia Palkina]

1. Create new environment (Ubuntu)
2. Choose Neutron, GRE
3. Choose Cinder LVM and Ceph for images
4. Add 1 controller, 3 compute+ceph
5. Start deployment. It was successful
6. Start OSTF tests.
Next tests failed on 1 step with error: Stack was not created properly. Please refer to OpenStack logs for more details.
- Typical stack actions: create, delete, show details, etc.
- Advanced stack actions: suspend, resume and check
- Check stack rollback
- Update stack actions: inplace, replace and update whole template

Logs are here: https://drive.google.com/a/mirantis.com/file/d/0B6SjzarTGFxab0tfOWktdTd2U1k/view?usp=sharing

"build_id": "2015-06-24_16-12-23", "build_number": "15", "release_versions": {"2014.2.2-7.0": {"VERSION": {"build_id": "2015-06-24_16-12-23", "build_number": "15", "api": "1.0", "fuel-library_sha": "7d19bc3783177aebf64fa4c2ae20d845cbd5348f", "nailgun_sha": "b74f847ec89c4bff1addb830704206dc503125f0", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-7.0", "production": "docker", "python-fuelclient_sha": "1b8574a7c4ea884862763a15c636b066d51f49e7", "astute_sha": "776157f722b13aff5f59bc098cf948793e6498ef", "fuel-ostf_sha": "69e7fa120e8efa7ed74d98efc63079d2f5c69d7b", "release": "7.0", "fuelmain_sha": "3b866d2ff3091a60362327028085fa62fd16c5a0"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "7d19bc3783177aebf64fa4c2ae20d845cbd5348f", "nailgun_sha": "b74f847ec89c4bff1addb830704206dc503125f0", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-7.0", "production": "docker", "python-fuelclient_sha": "1b8574a7c4ea884862763a15c636b066d51f49e7", "astute_sha": "776157f722b13aff5f59bc098cf948793e6498ef", "fuel-ostf_sha": "69e7fa120e8efa7ed74d98efc63079d2f5c69d7b", "release": "7.0", "fuelmain_sha": "3b866d2ff3091a60362327028085fa62fd16c5a0"

[Anastasia Kuznetsova (akuznetsova)]

After some log investigation it looks like heat-api worked good for a some time (in ostf-logs we can see, that at list one stack was created). After that something has happened, but there is no error in heat logs.

[Kairat Kushaev (kkushaev)]

Anastasia, could you please change the milestone if the issue is related mostly to Mirantis Openstack 7.0?

[Anastasia Kuznetsova (akuznetsova)]

@Kairat, firstly, we've need to figure out where the problem is: in heat puppet manifests, heat packages (than Fuel milestone is set correct) or somewhere in heat code (than here should be MOS milestone)

[Kairat Kushaev (kkushaev)]

Ok, I dig a bit into the problem.
The first fail for cluster_ostf_tests_platform tests shows the following:
2015-06-25 13:19:36 FAILURE Typical stack actions: create, delete, show details, etc. (fuel_health.tests.tests_platform.test_heat.HeatSmokeTests.test_actions) Time limit exceeded while waiting for stack creation to finish. Please refer to OpenStack logs for more details. File "/usr/lib/python2.6/site-packages/unittest2/case.py", line

AFAIU the timeout is 20 sec so the request to heat should be at 13:19:16.
The heat engine shows the following:
2015-06-25T13:19:30.967035+00:00 warning: WARNING: /usr/lib/ocf/resource.d/fuel/heat-engine: 1: kill: No such process
2015-06-25T13:19:30.972518+00:00 info: INFO: Old PID file found, but OpenStack Heat Engine is not running
2015-06-25T13:19:31.369649+00:00 warning: WARNING: /usr/lib/ocf/resource.d/fuel/heat-engine: 1: kill: No such process
2015-06-25T13:19:31.384361+00:00 info: INFO: Old PID file found, but OpenStack Heat Engine is not running
2015-06-25T13:19:31.395487+00:00 info: INFO: OpenStack Heat Engine is already stopped
2015-06-25T13:19:31.514685+00:00 warning: WARNING: /usr/lib/ocf/resource.d/fuel/heat-engine: 1: kill: No such process
2015-06-25T13:19:31.532960+00:00 info: INFO: Old PID file found, but OpenStack Heat Engine is not running
2015-06-25T13:19:31.589898+00:00 info: INFO: OpenStack Heat Engine started

Then heat engine restarts and works correctly around 1 min. After that there is no messages at all to heat-api and heat-engine which is strange.

haproxy tells the following:
Server heat-api/node-1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2015-06-25T13:20:51.611948+00:00 emerg: proxy heat-api has no server available!
 So heat-api and heat-engine are down after a minute.
I am wondering if it could be because there is not enough resource on controller but need to ensure.

[Kairat Kushaev (kkushaev)]

We tried to re-produce the bug with Anastasia but it was not reproduced.
It turned out that there were 2 deployments in parallel when the issue was re-produced.
So need to try to reproduce issue with parallel deployment.

[Anastasia Palkina (apalkina)]

Now I cannot reproduce it again.

If issue reproduce, I reopen this bug.

[Kairat Kushaev (kkushaev)]

The issue is not reproducing some time so I am moving this to Invalid.