ERR: (/Stage[main]/Neutron::Agents::Metadata/Service[neutron-metadata]/ensure) change from stopped to running failed: Execution timeout after 600 seconds!

[Anastasia Palkina]

This issue blocks us, as our new 3rd BVT build fails consistently. Last failure was marked as because of this bug.

"build_id": "2014-11-09_21-28-52",
"ostf_sha": "9c6fadca272427bb933bc459e14bb1bad7f614aa",
"build_number": "86",
"auth_required": true,
"api": "1.0",
"nailgun_sha": "8330f6221e190db87fc5baa735fa719c85a2e02d",
"production": "docker",
"fuelmain_sha": "f536d11fb40fed78c26f4d7326c45b0c48f157d8",
"astute_sha": "3c374c9f7bfbdbcd7ce2f716cd704e3044e6fb41",
"feature_groups": ["mirantis"],
"release": "6.0",
"release_versions": {"2014.2-6.0": {"VERSION": {"build_id": "2014-11-09_21-28-52", "ostf_sha": "9c6fadca272427bb933bc459e14bb1bad7f614aa", "build_number": "86", "api": "1.0", "nailgun_sha": "8330f6221e190db87fc5baa735fa719c85a2e02d", "production": "docker", "fuelmain_sha": "f536d11fb40fed78c26f4d7326c45b0c48f157d8", "astute_sha": "3c374c9f7bfbdbcd7ce2f716cd704e3044e6fb41", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "c7b71bd1ee939b5a634715ac7b13c2936ad93a5e"}}},
"fuellib_sha": "c7b71bd1ee939b5a634715ac7b13c2936ad93a5e"

1. Create new environment (CentOS, HA mode)
2. Choose GRE neutron
3. Choose Ceph for volumes
4. Choose Sahara, Ceilometer
5. Add 2 controller+ceph, 1 conroller, 1 compute, 3 mongo
6. Start deployment. It has failed
7. There is error on controller (node-5)

2014-11-10 17:41:39 ERR

 (/Stage[main]/Neutron::Agents::Metadata/Service[neutron-metadata]/ensure) change from stopped to running failed: Execution timeout after 600 seconds!

I reproduced this bug twice on this ISO. See logs.

Logs are here https://drive.google.com/a/mirantis.com/file/d/0B6SjzarTGFxaX0xMUFNiS2pGQXc/view?usp=sharing

[Alexander Nevenchannyy (anevenchannyy)]

Steps to verify:

At any controller node:
1) crm status

If you see error such this:

 Clone Set: clone_p_neutron-metadata-agent [p_neutron-metadata-agent]
     p_neutron-metadata-agent (ocf::mirantis:neutron-agent-metadata): Started node-1 (unmanaged) FAILED
     Started: [ node-2 node-3 node-6 ]

Failed actions:
    p_neutron-metadata-agent_stop_0 (node=node-1, call=190, rc=1, status=Timed Out, last-rc-change=Wed Jul 8 14:59:38 2015
, queued=30004ms, exec=0ms
): unknown error

2) crm resource cleanup clone_p_neutron-metadata-agent
3) crm resource managed clone_p_neutron-metadata-agent
4) Check crm status again

Metadata agents must be started at all controller nodes.