Deploy fails when radosgw_restart.sh executed

[Victor Ryzhenkin]

ISO: {"build_id": "2015-03-12_17-19-52", "ostf_sha": "ecb8e294b0acbdc5b0300d5e39028fb26ecc9088", "build_number": "191", "release_versions": {"2014.2-6.1": {"VERSION": {"build_id": "2015-03-12_17-19-52", "ostf_sha": "ecb8e294b0acbdc5b0300d5e39028fb26ecc9088", "build_number": "191", "api": "1.0", "nailgun_sha": "c186f71158ed27b03d8db87561ea66c19e39b452", "production": "docker", "python-fuelclient_sha": "59513d6b75f86060ff5059f39fdd9cca56c83f19", "astute_sha": "ed76b0cacf34a4a683b464ebd86e0beb273b5473", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "3764b8a73b3a93fd7ee66937ba4c4c77da409b78", "fuellib_sha": "fda8128b9ca7a8ce818421040f597a50eece8078"}}}, "auth_required": true, "api": "1.0", "nailgun_sha": "c186f71158ed27b03d8db87561ea66c19e39b452", "production": "docker", "python-fuelclient_sha": "59513d6b75f86060ff5059f39fdd9cca56c83f19", "astute_sha": "ed76b0cacf34a4a683b464ebd86e0beb273b5473", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "3764b8a73b3a93fd7ee66937ba4c4c77da409b78", "fuellib_sha": "fda8128b9ca7a8ce818421040f597a50eece8078"}

Steps to reproduce:
1. Deploy OS+neutron-gre, Murano, All Ceph, HA (3 Virtual controller, 1 hardware compute)

Actual result:
Deployment fails with error in Fuel-dashboard:

Deployment has failed. Method granular_deploy. Failed to execute hook .
---
priority: 300
fail_on_error: true
type: shell
uids:
- '3'
- '2'
- '4'
parameters:
  cmd: sh /etc/puppet/modules/osnailyfacter/modular/astute/restart_rados.sh
  timeout: 180
.
Inspect Astute logs for the details

In Astute:
[426] Error running RPC method granular_deploy: Failed to execute hook .
---
priority: 300
fail_on_error: true
type: shell
uids:
- '3'
- '2'
- '4'
parameters:
  cmd: sh /etc/puppet/modules/osnailyfacter/modular/astute/restart_rados.sh
  timeout: 180
, trace:
["/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/nailgun_hooks.rb:54:in `block in process'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/nailgun_hooks.rb:26:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/nailgun_hooks.rb:26:in `process'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/deployment_engine/granular_deployment.rb:219:in `post_deployment_actions'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/deployment_engine.rb:74:in `deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/orchestrator.rb:118:in `deploy_cluster'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/orchestrator.rb:56:in `granular_deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/dispatcher.rb:99:in `granular_deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/server.rb:142:in `dispatch_message'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/server.rb:103:in `block in dispatch'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/task_queue.rb:64:in `call'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/task_queue.rb:64:in `block in each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/server.rb:101:in `each_with_index'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/server.rb:101:in `dispatch'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/server.rb:85:in `block in perform_main_job'"]

Snapshot will be attached via Dropbox ASAP .

[Victor Ryzhenkin (vryzhenkin)]

Snapshot: https://www.dropbox.com/s/lszna41u7c3v0iq/fuel-snapshot-2015-03-13_11-00-51.tgz?dl=0

[Timur Nurlygayanov (tnurlygayanov)]

The importance was changed to High because deployment failed with Ceph RadosGW.

[Vladimir Kuklin (vkuklin)]

Do we have this issue reproducing right now? I saw it only once. Let's mark it as incomplete as our CI shows RadosGW tests passing right now. Also it is unclear which OS was used.

[Victor Ryzhenkin (vryzhenkin)]

Vladimir, it's ok now.
All ceph deployed correctly with 254 iso.

[Victor Ryzhenkin (vryzhenkin)]

CentOS based.