Controller becomes offline after deployment for CentOS

[Anastasia Palkina]

"build_id": "2015-01-27_22-55-01",
"ostf_sha": "c9100263140008abfcc2704732e98fbdfd644068",
"build_number": "82",
"release_versions": {"2014.2-6.1": {"VERSION": {"build_id": "2015-01-27_22-55-01", "ostf_sha": "c9100263140008abfcc2704732e98fbdfd644068", "build_number": "82", "api": "1.0", "nailgun_sha": "7a3d06074c18f90b46000468d49fe8624cd2c519", "production": "docker", "python-fuelclient_sha": "1f3ba542dd93cf79fa5ec92880fc1fbde4c97e00", "astute_sha": "ed5270bf9c6c1234797e00bd7d4dd3213253a413", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "", "fuellib_sha": "abcccb60c1e6bde225ede10992fa123d3e092015"}}}, "auth_required": true, "api": "1.0", "nailgun_sha": "7a3d06074c18f90b46000468d49fe8624cd2c519", "production": "docker", "python-fuelclient_sha": "1f3ba542dd93cf79fa5ec92880fc1fbde4c97e00", "astute_sha": "ed5270bf9c6c1234797e00bd7d4dd3213253a413", "feature_groups": ["mirantis"],
"release": "6.1",
"fuelmain_sha": "",
"fuellib_sha": "abcccb60c1e6bde225ede10992fa123d3e092015"

1. Create new environment (CentOS)
2. Choose neutron, gre
3. Choose Ceilometer
4. Add 3 controller+mongo+cinder, 2 compute, 1 cinder, 2 mongo
Controllers have 3072 Mb of memory and 2 CPU
5. Start deployment. It was successful
6. Start OSTF tests.
7. Second controller became 'offline' and this node is not available and virtual machine hangs

I reproduced it for CentOS 3 times

Logs are here: https://drive.google.com/a/mirantis.com/file/d/0B6SjzarTGFxaRTY0NHZkT0s2QUk/view?usp=sharing
Offline controller is node-6

[Anastasia Palkina (apalkina)]

I reset node-6 to collect logs from it. They are above

[Oleksiy Molchanov (omolchanov)]

was fixed by ceilometer team.

[Anastasia Palkina (apalkina)]

It reproduced again on ISO #92

"build_id": "2015-02-01_22-55-01", "ostf_sha": "c9100263140008abfcc2704732e98fbdfd644068", "build_number": "92", "release_versions": {"2014.2-6.1": {"VERSION": {"build_id": "2015-02-01_22-55-01", "ostf_sha": "c9100263140008abfcc2704732e98fbdfd644068", "build_number": "92", "api": "1.0", "nailgun_sha": "c0932eb5c2aa7fd1e13a999cb1b4bf5aff101c3b", "production": "docker", "python-fuelclient_sha": "2ea7b3e91c1d2ff85110bf5abb161a6f4e537358", "astute_sha": "ed5270bf9c6c1234797e00bd7d4dd3213253a413", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "", "fuellib_sha": "c5e4a0410ba66f9e9911f62b3b71c0b9c29aed6e"}}}, "auth_required": true, "api": "1.0", "nailgun_sha": "c0932eb5c2aa7fd1e13a999cb1b4bf5aff101c3b", "production": "docker", "python-fuelclient_sha": "2ea7b3e91c1d2ff85110bf5abb161a6f4e537358", "astute_sha": "ed5270bf9c6c1234797e00bd7d4dd3213253a413", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "", "fuellib_sha": "c5e4a0410ba66f9e9911f62b3b71c0b9c29aed6e"

1. The same environment was deployed successfully
2. Controllers were online
3. When I verify other bug I killed rabbitmq processes on node-6 and waited until rabbitmq restarted. it was successful
4. But after some minutes other controller (node-7) became offline and virtual machine hangs, no connection
5. I restarted node-7 so you can see logs from this node

Logs are here: https://drive.google.com/a/mirantis.com/file/d/0B6SjzarTGFxaTHF5X2RWbUdqNlE/view?usp=sharing

[Oleksiy Molchanov (omolchanov)]

I didn't manage to find something meaningful in logs, can someone check?

[Anastasia Palkina (apalkina)]

I inreased the RAM on controllers to 5 Gb.

Issue in this case not reproduced.

I move this bug to Incomplete status because the specific cause is not identified.

[Stanislaw Bogatkin (sbogatkin)]

This bus setted as incomplete for 4 weeks now. Setting as 'invalid'.